-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOP-11752] - add support of incremental strategy in Kafka #202
Conversation
e71ab2c
to
e20944b
Compare
не нашел в каких случаях "в разные моменты времени в window.start_from.including может быть true, а может быть и false", если мы используем только |
e20944b
to
9d15f37
Compare
9d15f37
to
eb30fbf
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #202 +/- ##
===========================================
+ Coverage 93.86% 94.11% +0.24%
===========================================
Files 204 204
Lines 7780 7799 +19
Branches 1379 1387 +8
===========================================
+ Hits 7303 7340 +37
+ Misses 358 335 -23
- Partials 119 124 +5 ☔ View full report in Codecov by Sentry. |
eb30fbf
to
8507fa2
Compare
также не нашел в каких случаях может передаваться |
...ration/tests_incremental_batch_strategy_integration/test_strategy_incremental_batch_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
tests/tests_unit/test_db/test_db_reader_unit/test_kafka_reader_unit.py
Outdated
Show resolved
Hide resolved
8507fa2
to
fe1f8cf
Compare
fe1f8cf
to
893d74f
Compare
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Show resolved
Hide resolved
Да, false может быть только в случае Batch стратегий, которые Kafka не поддерживает. Можно добавить комментарий об этом код формирования
Для остальных СУБД это используется в случае, когда в источнике нет строк, попадающих под where, и min/max возвращают В случае Kafka даже у пустого топика или топика без новых данных есть какой-то offset для каждой из партиций, и |
Еще нужно проверить документацию на наличие предупреждений о работе Kafkaс инкрементами: onetl/docs/connection/db_connection/kafka/read.rst Lines 8 to 10 in b4ba949
|
Сделай rebase, я пофиксил сборку документации |
поправил |
…remental strategies
67bfbea
to
0f3d812
Compare
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
7a5b537
to
13d297c
Compare
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
13d297c
to
a20fb58
Compare
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
...strategy_integration/tests_incremental_strategy_integration/test_strategy_increment_kafka.py
Outdated
Show resolved
Hide resolved
9a6f524
to
1f2ea3e
Compare
Change Summary
Incremental Strategies
forKafka
connection*
and,
toKafka.source
as we handle reading only from single topic (inDBReader
concept +HWM
support)Related issue number
Checklist
docs/changelog/next_release/<pull request or issue id>.<change type>.rst
file added describing change(see CONTRIBUTING.rst for details.)